Ship the CUDA delegate in the wheel - #21645
Open
shoumikhin wants to merge 1 commit into
Open
Conversation
Contributor
Author
shoumikhin
requested review from
abhinaykukkadapu,
digantdesai,
kirklandsign,
larryliu0820,
mergennachin and
psiddh
as code owners
August 7, 2026 15:49
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/21645
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New FailuresAs of commit 3fa305d with merge base cc853ae ( NEW FAILURES - The following jobs have failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
This was referenced Aug 7, 2026
|
|
The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.
There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.
Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.
```cmake
find_package(executorch REQUIRED COMPONENTS backend_cuda)
target_link_libraries(my_app PRIVATE executorch::runtime
executorch::backend_cuda)
```
A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:
```
executorch/lib/libexecutorch_backend_cuda.so the delegate
executorch/lib/libexecutorch_extension_cuda.so the stream helper
executorch/backends/cuda/libaoti_cuda_shims.so the GPU device code
```
Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.
The declared set includes the runtime compiler, because a shipped library links it to build kernels
at run time. Each declared package also needs its own directory recorded, since that is where the
loader looks. On the CUDA 12 packaging the compiler installs into its own directory, and omitting it
left that library unable to find the compiler even though the package was installed. The CUDA 13
packaging puts every component in one directory, so the same gap does not appear there.
The stream helper's header no longer includes `cuda_runtime.h`, which the wheel does not publish. It
only ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer
can compile against the wheel with no CUDA toolkit installed.
Built a CUDA wheel, installed it into a clean environment, and:
- ran a GPU model from Python on an NVIDIA GPU, with output identical to eager PyTorch on the
same weights and inputs (largest absolute difference 0).
- built a C++ application against the installed wheel alone and ran the same model, matching the
same reference.
- confirmed one library defines the stream state and the GPU shims, not several. Extracting them
into every consumer put three copies in one wheel, and a stream selected through one was invisible
to the others.
- confirmed no shipped library records a CUDA toolkit path from the build machine, and every library
that links the CUDA runtime has a relative path to it.
- confirmed a CPU wheel ships none of the CUDA libraries and no CUDA-only header.
- a row that names a CUDA train is built with the CUDA option on rather than left to autodetection,
so a builder without a matching toolkit fails while configuring. Before this, such a row produced a
wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it
installed cleanly and then reported the backend as unregistered when a model ran.
- a row whose major CUDA version does not match the installed toolkit now fails the build. The
declared packages and the loader paths come from the row while the binaries come from the
toolkit, and nothing compared the two, so a `cu126` row built against a 13.0 toolkit attached
CUDA 12 metadata to binaries needing `libcudart.so.13`. An unrecognised train fails too, instead
of silently reporting whatever the builder happened to have. Detection reads the toolkit major
directly, so the guard fires on any mismatch rather than only on the three exact `(major, minor)`
pairs the supported list carries; on those three pairs it behaved correctly before, and on every
other minor it saw an empty detection and skipped the check.
- the row classifier and the packaging read the row the same way now, so both agree on what a row
spelled with an unsupported minor means. The shell classifier reduces the row to digits and
matches against `SUPPORTED_CUDA_VERSIONS`. Packaging did the same shape on the outer decision
and then took only the first two digits when picking runtime packages, so `cu125` classified as
CPU on one side and declared CUDA 12 on the other. Packaging now matches on the same digits and
raises loudly on an unsupported train instead.
- whether a row is a CUDA row is decided by asking if it names a supported train, rather than by
listing the spellings that mean "no CUDA". Checked 16 row values including `cpu-aarch64`, `rocm6.2`
and `cu118`; the previous list-based form was wrong on several, and each wrong answer made a
non-CUDA wheel declare the CUDA runtime.
- the CUDA components are required when the wheel's own version says it is a CUDA wheel. They were
optional unconditionally, so a wheel tagged `+cu126` with no CUDA library at all passed every check.
- the stream helper ships under either name it can be built with. The shim layer records it as a
dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a
non-shared build shipped a shim whose dependency resolved to nothing.
- the relative hops between shipped libraries are sized by how deep the library sits in the package.
A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops
landed on a directory that does not exist, and the hop from `lib/` climbed out of the package
entirely, where an unrelated library with a matching soname could satisfy the dependency first.
- the stream helper no longer links or includes the CUDA toolkit. It uses a stream only as an
opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link.
Also fixed in this commit:
- The pre-build classifier resolves the Python interpreter (`python3` or `python`, whichever
exists) instead of assuming one name, and no longer discards stderr. Builders disagree on the
name: Linux and macOS provide `python3`, while the Windows builder runs inside a conda
environment that provides only `python`. Assuming either name breaks the other platform, and
treating the failure as "not a CUDA row" silently rebuilt a CUDA row as a CPU row.
- `CU_VERSION=cpu pip install .` is handled explicitly instead of running the CUDA-train parser
over it, which previously turned `cpu` into `pu` through a character-set strip and reached the
unsupported-train error.
Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.
Known gap, not introduced here: the Python `Runtime.load_program` path allocates activation memory
on the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.
ghstack-source-id: ff24a2d
ghstack-comment-id: 5219161655
Pull-Request: #21645
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CUDA delegate runs a model on an NVIDIA GPU. It is built into the Python extension, so only
Python can use it. A C++ application has no way to link it, and nothing else can reuse it either.
There is a sharing problem too. A program may use more than one GPU backend at once, and they need
to agree on which CUDA stream (the queue the GPU runs work on) the caller chose. If each backend
carries its own copy of that state, work queued through one is invisible to the other.
Ship the CUDA delegate and a small stream helper as their own shared libraries, and name both as
CMake components. The stream helper is shared so a process has exactly one copy of the caller's
stream choice, which is what lets two backends agree on it.
A CUDA wheel does not bundle the CUDA runtime. It declares it as a dependency, the way the PyTorch
CUDA wheels do, so one copy is shared with torch rather than shipping a second one:
Each library records a relative path to where pip installs the CUDA runtime, so it resolves without
the caller setting a library search path and without depending on a toolkit being installed.
The stream helper's header no longer includes
cuda_runtime.h, which the wheel does not publish. Itonly ever uses a CUDA stream as an opaque handle, so it declares that handle itself, and a consumer
can compile against the wheel with no CUDA toolkit installed.
Built a CUDA wheel, installed it into a clean environment, and:
same weights and inputs (largest absolute difference 0).
same reference.
into every consumer put three copies in one wheel, and a stream selected through one was invisible
to the others.
that links the CUDA runtime has a relative path to it.
so a builder without a matching toolkit fails while configuring. Before this, such a row produced a
wheel tagged for CUDA, carrying no CUDA library, that still declared the CUDA runtime packages: it
installed cleanly and then reported the backend as unregistered when a model ran.
declared packages and the loader paths come from the row while the binaries come from the
toolkit, and nothing compared the two, so a
cu126row built against a 13.0 toolkit attachedCUDA 12 metadata to binaries needing
libcudart.so.13. An unrecognised train fails too, insteadof silently reporting whatever the builder happened to have. The comparison is at major only, so a
cu126 row built with a 12.8 toolkit still passes.
listing the spellings that mean "no CUDA". Checked 16 row values including
cpu-aarch64,rocm6.2and
cu118; the previous list-based form was wrong on several, and each wrong answer made anon-CUDA wheel declare the CUDA runtime.
optional unconditionally, so a wheel tagged
+cu126with no CUDA library at all passed every check.dependency whenever CUDA is on, while packaging named only the shared-build spelling, so a
non-shared build shipped a shim whose dependency resolved to nothing.
A fixed pair was correct at one depth only: measured over every shipped location, 6 of 12 hops
landed on a directory that does not exist, and the hop from
lib/climbed out of the packageentirely, where an unrelated library with a matching soname could satisfy the dependency first.
opaque handle and calls no CUDA function, so it needs no toolkit include and no libcudart link.
Also fixed in this commit:
python3rather than a barepython, and does not discardstderr. On a host without a
pythonalias any CUDA row was silently rebuilt as a CPU row.CU_VERSION=cpu pip install .is handled explicitly instead of running the CUDA-train parserover it, which previously turned
cpuintoputhrough a character-set strip and reached theunsupported-train error.
Ran end to end on H100, A100 and Jetson Thor, covering compute capabilities 9.0, 8.0 and 11.0.
Known gap, not introduced here: the Python
Runtime.load_programpath allocates activation memoryon the host, so a program exported to keep activations on the GPU fails there. The supported Python
loader and the C++ path both work. This is upstream in the Python bindings, which this change does
not touch.